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ABSTRACT 

A study involving a simple random sample of 180 
county-level administrators employed by the U.S. Cooperative 
Extension Service assessed the effects of client and administrator 
involvement as stakeholders in evaluation. A posttest-only control 
group design was used. Variables included level of administrator 
involvement as stakeholc^iirs and level of client involvement as 
stakeholders. Each subjfi'Ct was assigned to one of nine treatment 
groups, reflecting ninf.- permutations of a simulated evaluation 
report. The instrument was administered via the mail. Three mailings 
cf the questionnaire booklet and follow-up postcards resulted in a 
final response rate of 94%. Results indicate that: (1) involvement of 
administrators had no direct effect on administrators' readiness for 
evaluation utilization; (2) perceptions of technical adequacy, 
increased stakeholder participation, and the perceived quality of the 
report were substantially related to utilization readiness; (3) 
attitude toward evaluation and readiness for evaluation utilization 
were significantly related? (4) the relationship between attitude 
toward evaluation and training in evaluation was negligible; and (5) 
d significant portion of the variance in readiness for utilization 
was explained by the variables in the generated model. Two flowcharts 
and one bar graph are included. (TJH) 
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Introduction 

Evaluation theorists and practitioners of program evaluation have been 
grappling for years with the concept of utilization of evaluation results. 
What constitutes utilization? What factors are involved in the utilization 
of evaluation results for decision itaking regarding program improvement? 
Hew can these factors be investigated? How can these factors be optimized? 
Stakeholder participation in the evaluation process has emerged as ?n 
important factor, and has been addressed by theorists and practitioners in 
the recent literature (Alkin, 1985; Ayers, 1987; Greene, 1987 and 1988; 
Patton, 1988; Snith, 1988; Wteiss, 1983). Gold (1983) defined stakeholders 
as people whose lives are affected by the program and people whose decisions 
can affect the future of the program. Meaningful participation has been 
defined as shared decision making, wherein stakeholders are considered to be 
active "collaborators in inquiry" (Stake, 1983:18). 

Greene (1988) reported several case studies that investigated the link 
between stakeholder participation in the evaluation process and evaluation 
utilization, in her wt^rds, "Data from these studies provide support for the 
value and viability of stakeholder participation in evaluation as one route 
to increased utilization of evaluation findings" (p.lOO). She documented 
both enhanced "readiness for utilization' on the part of decision makers and 
actual use of evaluation results (instrumental, conceptual and syntsolic 
use). "Readiness for utilization" (Greene, 1988) is a construct which 
deals with the contributions of the participatory eveduation process to 
results utilization. The construct includes 1) ^rceptions of the validity 
and credibility of results, 2) understanding of results, 3) acceptance of 
results and 4) sense of responsibility to follow through on the results. 
These variables appear to be directly related to those addressed by the 
Utility Standards (Joint Committee on standards, 1981). It is suggested 
that as the values of each of these four variables increase, the chances for 
subsequent utilization (however it is defined) inprove. In addition, a 
variety of human, context and evaluation factors (Alkin, 1985) are expected 
to be related to readiness for utilization— these include a users' 
pre-existing attitude toward evaluation and several variables related to the 
users' perceptions of the report, such a.« perceived quality of the 
evaluation report and perceived technical adequacy of the evaluation. 

While an overview of the literature on utilization of evaluation 
suggested that evaluation results in education are seldom used, little or 
nothing is known about utilization within the U.S. Cooperative Extension 
Service (CES) in particular. The CES provides a wide variety of educational 
programs in agriculture, home economics, natural resources and comnunity 
development for adults and youth in the United States. Evaluation of 
individual programs at the county level is undertaken largely with the aim 
of improving programs in the future. A recent ECOP report (Futures Task 
Force, 1987:14) points out, however, that "no mechanism exists to facilitate 
the use of evaluation results in making decisions, reaching conclusions or 
forming jud^nents about the effectiveness of Extension programming". Yet 
county administrators (e.g. county directors, county chairpersons) in the 
cooperative Extension Service are key decision makers as the intended users 
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of a variv-«:y of program evaluation results. Hieir participation as 
stakeholders and/or their perceptions regarding the participation of clients 
and program staff as stakeholders may have a direct effect on the 
utilization of those evaluation results. 

The purpose of the study 

The problem addressed by this study involves stakeholder involvement in 
the evaluation process and whether perceived involvanenc is related to 
county administrators' readiness for utilization of evaluation results in 
the context of the U.S. Cooperative Extension Service. 

The major research question of this study can be stated: How do two 
factors known to vary in U.S. CES program evaluations (level of involvement 
of clients as stakeholders and level of involvement of administrators as 
stakeholders) relate to county level decision makers' readiness for 
utilization of the evaluation results? A model of readiness for results 
utilization developed for this study (Figure 1) illustrates the expected 
relationships among the variables under investigation. The various 
geometric shapes represent individual variables or sets of variables and 
arrows indicate expected significant correlations, stakeholder involvement 
in CES evaluations is one factor that could be optimized if more were known 
about the relationship between stakeholder involvement and readiness fcr 
utilization of results by county administrators. 

I'he ultimate aims of the study were to provide einpirical research 
results that may contribute to the development of a general model of 
utilization of evaluation results, and to determine whether involvement of 
stakeholders plays a key role in the readiness for utilization of evaluation 
results in the U.S. CES. Readiness for utilization has been stated to 
represent "the core contribution of the participatory evaluation process to 
subsequent results utilization- (Greene, 1938). The findings will aid the 
evaluation profession as a whole, as well aii evaluation personnel in CES to 
better understand the concepts involved in readiness for utilization, and 
will serve as a guide for evaluation practitioners who seek to improve the 
utilization of program evaluation results. Recomnendations were developed 
for three groups known to be concerned with the utilization of 
evaluation— evaluation practitioners, extension educators and evaluation 
researchers. 

Objectives of the study 

* To determine the effects of varying a) level of involvement of 
administrators as stakeholders (ADMIN) and b) level of involvement of 
clients as stakeholders (CLlwr) within a simulated evaluation report upon 
administrators' readiness for utilization (UTIL) of the evaluation results. 

* To investigate reported and/or hypothesized relationships among 
readiness for utilization of results and the following factors in order to 
confirm/ disconfirm these as they apply to utilization of evaluation results 
in the CES. 

a. Administrator attitude toward evaluation (ATT) 
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Rgura 1: A Model of the Factors Influencing Administratoi^' Readiness for 
Utilization of Program Evaluation Results in the U.S.CES. 
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r ^ Manipulated variabtet: le'tdi of diant involvement as stakeholders (CLIENT) 
level of admintttrator involvement as stakeholders (ADMIN) 

I I Set of variables related to perceptions ol the report: technical adequacy (TECH). 
' ' quality (QUAL) and adequacy of stakeholder paiticipatbn (STAKE) 

r\ Antecedent variiibles: administrator experience (EXP), tninino in evakjatk)n (EVAL). 

previous use ot evakjation results (USE) and attitude toward evakjation (ATT) 

Ultimate dependent variable: admMstratof's readl. less for utization (UTIL) 
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b. Actainistrator training in evaluation methods (eval) 

c. Professional experience as adninistrator (EXP) 
a. Previous use of evaluation results (USE) 

e. Perceived technical adequacy of the evaluation (tech) 

t. Perceived quality of the evaluation report (qual) 

g. Perceived adequacy of stakeholder participation (stake) 

/»n„t„'^^^®"I'^"®.^'^® relative iinxjrtance of the manipulated variables 
(ADMIN, CLIENT) and measured variables (ATT, STAKE, QUAL, TECH) in 
accounting foe the variance in readiness for utilization of results (UTIL) 



Methodology 



The target population was county level administrators in the 3050 
counties served by the U.S. cooperative Extension Service. The sample used 
^5"<^yv«s a simple randan sample of 180 CES county administrators, a 

aiiaAc^^ I Edition) county Agents, a reference directory for 

extension workers was used as the frame for the study. 

the DSt^S^iS^^^Ji/?""^ was conducted using a research design known as 
dSi^Sitf L^''f°iJ''°ff 5^'^- specifically, a two-way factorial 
design with •-.wo manipulated, fixed, crossed independent variables was used 

SeS^ll^.'^i^^"'?' variables (level of adniSstrator ii?SlvlmS? S 
wfS^ifi^f.f^ ^^"^^ °^ involvement as stakeholders), had three 

lariiinff i^c'^J:? E«""tations of a simulated evaluation report, l^e 
trea^Ht g?Spf™ randomly assigned to one of the 9 

The threats to internal validity of the study are reduced by the 
research design. In addition, the measurement of a variety of potential 
intervening variables (antecedent variables and variables relatS to 

i^''^S^^S?^fc''l;L''^''^^J^^'*^ researcher to use these as covariates 
m the analysis and provide an increase in the power of the significa ce 
f?fj ifff^ similar to that provided by a pre-tesf (Campbell and stanl. /, 

r^r«,SfSf^.f° validity include the possible interaction of 

S variables and treatment effects, and reactive effects, such 

^^r^ ^Canpbell and Stanley, 1963:8; Bracht i Gl«Js, 

llliil^ !5^«?^®**^,?L^"?®"^^°" ^ the measurement of a 

^«fiUf f^^^^^<^J^^i^^^^, allowing the researcher to investigate 
possible interactions with the treatment. iSe threat of reactive effects, 
which appear when participants change their behavior in response to their 
Kl^i?".®^ involved in an experiment, was reduced by the fact that 

^^Sf'^^^'J"^? "^^^ '^^^ uninformed as to which of the inSpendent 
S^iSnf^i" ?® ^^^^ manipulated and which were controlled. The 
^ibility of non-response error was reduced by repeated follow-ups by 
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The treatments were a series of vignettes representina a model Droaram 
evaluation report for an agricultural extension p?4ram . ThI ?Sctf Srf 

involvement (the manipulated variables). 

The first manipulated independent variable was a construct of factors 
including evaluator's willingness to involve administrators as usersf 
^^«^?nJ.^ amount/quality of information dialogue during 

altf^eiSdir"- wIS^^"f^"2 was labelled -involvement of adminiltratcr al 

i^S %hr2^i22?^'^''^ ordinal and was identified as admin, -niis 
involved levels; not involved, moderately involved and highly 

The second manipulated independent variable was the construct 
representing "level of involvement of clients as stakeholders". This 
SS^SiL'?^ considered ordinal and was identified as aiEWr. This variable 
had three levels; not involved, moderately involved and highly involved! 

Construct validity for the treatments was assessed by a oanel of 

correlations betwel? int^dS JeSS^ of the 

lUiiEOTj and 0.83 for administrator involvement (ADMIN). 

variables were either measured for each individual in the 
study (Experience as an administrator~EXP, training in evaluation--EVAL 
IJSfa^on"^.?5 re«.lts~uSE, pre-existiS^ StuSS ti2Srd ' 

r Pfi^^ived adequacy of stakeholder involvement-CTAKE, 
t^l ri^r.^r"'^/^?^ °^ evaluation-TECH, perceived quality of 
ell^riS^StT^^ readiness for utilization-OTIL) S controlled in the 

Cronhi!;ht«"!U?}^- -^J? °^ sunnated scales was determined by using 
?a^!j^Ji"o%f^i^ S;?^ ^^-^^^ ^th OCES personnel. RelSbilit? 
values ranged from 0.71 for STAKE to 0.89 for ATT. 

utili2??^"2«''fiifi^^°^^"^'^"* ^"^'^ evaluation and tor readiness for 
in in J^?i,V?^.!?®2f*' '/^^'^ ^ ^°"P^ °^ OSU graduate students eni'olled 
J ?fUj**'?''^°" ^^^^^^ methods course. Correlations (r phi) 
SS^^o?1hi mJ^? ^P^^'^^^^ ""^'^^ "^tive) and the Measured 

ISlT(Sr) 0 determined to be 0.95 for the attitude 

scale (ATT) and 0.92 for the readiness for utilization scale (OTIL). 

Data collecti<m 

m.n UlfJf '"^T"'' °L'»asurement was provided to the participants as a 
SSiiiS^SJJS!!^^''? questionnaire was designed as fs.S inch 
booklet which included instructions on low to cSplete the questionnaire 

^onSS!uo'S^."Lr'S: r'^^yfii^? ^^^f"«- ^ mailin^^of ihS^klet 
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Descriptive statistics were generated and presented for the oersonal 

°5 administrators. All the^analysS Tf^, ^l^Tlere 
performed using the Statistical Package for Social Science (spss/pc^)? 

The data were assessed in order to detect any violations of the 
SJ^Hnf distributions of the inde^ndent 

Tn^SiS ^ °! ""Iticollinearity. Residuals were Assessed for 

independence, absence of correlation with the independent variabler 

^l^oar^l distribution. A series S ?S essiJf ^^tions 
S^!ai^ thf SJhS^ -"'"^ a hierarchical model, in orSr to att^t to 
explain the variance in -eadiness for utilization of results (otil). 

Findings and Discussion 

The respondents scored a mean of 2.92 (of a Dos<5ible d on) unm a 

feSr?2 T^^'Tf °" ^"^^"^^ sclle^rm)''' ^Res^n^^Sts 

^(^) ^Hpn;i r^IJ f experience as county directors in the U.S. 
OS (EXP). Experience ranged from as little as one year ud to 32 vears 
^th a standard deviation of 7.65 years. The meS^ nSr^f evaluation' 

past by the respondents was eight (standard deviation-8.4) . Reoardina the 

S^n?3 nine I^nili.?" ^ SL^ ^ "^^'^ ^-^^^^ S-frequently. 

^h^^^|. „2! ^^^"iftrators responded to an open-ended question req^rdina 

(Svif ImSrovr^^r?. ?" "^'^ «^ fo^^SiSng 

betweS^irSL°wJi'^'^2^ '^^^'^ ""^^'^ "^'^ "° significant differences 
oetween groups in ATT, EXP, EVAL or USE. These findings (the lack of 

seJ2t'!S?^n5^Jjn^'° antecedent variables) support thi use of rarJ^ 
selection and randan assignment as the best means for obtainina tS nine 
equivalent treatment groups used in the experiment. "® 

stakeS^?i"?iS£jSf i''! ^' °^ administrator involvement as 

S tS^rSSfti^M^^^-riS on readiness for utilization 

?8Li^f Sh f i^^h ^ ^^''f} °^ ^^^^^'^ involvement as stakeholders 
wSe^ ?^?vfi^l ISf'' non-linear effect on OTIL. When administrators 

S^J^f ?® evaluation process, the quadratic function 
explained a significant amount (9.1 percent) of the variance in OTIL. 

ainh;,iJ''??^!'^ °^ 9^^^ "^'^ Statistically significant at 

alpha-0.10, only a small portion of the variance in the dependent variable 
Sf ^Sf^?^' the practical significance of thTfinSSgs SSw 

S?2n??«^!J'f ~' as stakeholders did the iS?oiveSnt of 

^rff.^.^''^®^^*^''^ practical significance, when 

adninistrators were not involved in the evaluation, a midewS level of 
Jiin?? i"volvanent as stakeholders was related to ^rceptions of increased 
technical adequacy (TECH), increased adequacy of sScSSder p^rtici^?iS 



Figure 2 sunmarizes the findings relatec to the hvDothesized 
JHi^;^"f^^^ ^'^"'^ (Objective 2)-the ?hIckne^o? Itelrrows in 

^^S'^S/oS^f n^iJ'tnf relationships between the"«iaSes. 
aSS^^A^? aS i^nl, f Si^i^fif^^'^^?"^.^'^^ '^^'^ variables TECH, QUAL 
Silfl^of rS?ir\ Jr^Si'^^S^® relationship with readiness for 
^th ^liT.^T^Ii>l; S^f^ ^ «3uared had low positive correlations 

SeSaU^hf l^ii.;^ low negative correlations with tech and Q(iAL. 
S^Sinii;/,^ correlations between the manipulated variables and the 

Sfe^JSatf^'^iS.^^J^ ^"7/ ^2^^' Alternative treatments used 
for evaluation research could allovf administrators to play more active roles 

Suf^^iln?^ ^ "^^"9 the tele^^e? a re^ar^hl? 

SSJt iSn^n«^,^^^"iJ''"''°' °^ ^ °"9o^"g evaluation, ask for input or 
^SJnSr'^H^n ^'/"^ 5?® ^'^"'^^gy "^ght produce a ^Sronger' 

tSrSist^Jr rnJ? f ''^^'^'^"'^ ^^'^^'^ expecting 

the administrator to role play involvement in the evaluation retroactively 

•'^fiS'ff ^ positive relationship with evaluation 

training (eval), while EVAL had a low positive correlation with previous use 
?L^r3^Sit°"/^'"J'li"^>' ^ negligible relationsMp Sth attltSSI 
^ i'^^^- ^® l^^l^ °f 3ny significant relationship betwLn 

f^uJ??nn'^ particular interest, sin?e inservice educati^S 

evaluation had been considered as a possible route to improved attitude 
toward evaluation. Previous use of evaluation results (USE) had a moderate 
tSriS^ulE '"^'^ P^^'^P^ ^ has its effect on 

uMnJ^Hnll^riiJT^ ^Jf^ positive relationship with readiness for 
reiationsSJn^^ J'i Sir^® ^ad substantial positive 

Ind OT?L ind ^^rr^® relationships between CLlEtrr and TEcS, CLIEOT 

and UTIL, and ATT and OTIL were statistically significant only when 
administrators are not involved as stakeholders in the evaluation 

Objective 3 concerned the relative importance of the variables in th- 
"1^^ M.?f^ !^ variance in readiness for utilization, overall, 

aiSr ?[?S5^2i.r2S'^.S^ ^SfSI"^"' ^^"^^^^^^ the regression model 
S W^'v^iSl^^!?;/^' f^'.?^^ and TECH accounted Sr 51.4 percent 
?LoVvJ f««?}"«ss for utilization. When administrators weS not 

S^i^nJi, J® P^'^^t of the variance in OTIL could 

Tf l^lf variables, however, att alone accounted for more 

of the variance (for an increment of 16.5 percent). 

n«rr«I5®!!/?l!li"^^''f^''°".*^'^® moderately involved in the evaluation 61.1 
Ss^SL L,Srt;;?t^^J;?.r^ ^ accounted for by these variables, ATT 
^ril^ ^ If^ P^^'y^ ^ ^""3^^ "^ole in acounting for the 

dfd^r;^n?J^,?f^?^^^If''°",*^'^® highly involved in the evaluation, ATT 
did not contribute significantly, while STAKE, TECH and QUAL together 
accounted for almost 68.0 percent of the variance in readiness for 



ADMIN 




STAKE 

TECH 

QUAL 

(s«l ol rtport variabiM) 






Readiness for 




UtiliZiftlion 








UTIL 



Figure 2: Strength of the Correlations amonp the Factors Influencing 

Administrators' Readiness for Utilization of Program Evaluation Results in the U.S. CES. 

CZ^ Manipul«ed variablM: level o« diem invotvement as stakeholders (CLIENT) 

level of administrator mvotvemem as stakeholders (ADMIN) 

I I ^ *" ''^Sa^fL'fS'f P*"^"™ "» foPort; lechnical adequacy (TECH). 

<niality (QUAL) arxl adequacy o«8takehokterpafticip.tiori (STAKE) 

O ^*'tl^„r.VJ^'"-«*"'"'«"'0' «P«»nce (EXP). trainir>g In evaluation (EVAL). 

previous use ol evaluation results (USE) and attitude toward evahjatnn (ATT) 

Ultimate dependent vanaWe: administratof s readiness for utilizaton (UTIL) 
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utilization (Figure 3a). In genoral, as the involvement of administrators 
as stakeholders increased, pre-existing attitude toward evaluation had less 
influence, and perceptions of the adequacy of stakeholder participation, 
guality of the report and technical adequacy of the evaluation had more 
influence on administrators' readiness for utilization of the evaluation 
results. Oily when adninistrators were not involved in the evaluation did 
the involvement of clients as stakeholders have any practical significance 
(CLIENT and CLIENT squared accounted for 9.1 percent of t\ . variance in 
UTIL). 

Siirilarly, when clients were not involved as stakeholders, ATT 
accou.ited for an 11.6 percent of the variance in OTIL, while when clients 
were moderately or highly involved, the contribution of ATT dropped to 5.2 
percent and 3.2 percent respectively (Figure 3b). in general, whenever 
administrators and/or clients are not involved as stakeholders in the 
evaluation, the pre-existing attitude of the administrator toward svaluation 
played a more important role in his/her readiness for utilization of the 
results, whereas when stakeholders were involved, STAKE played a more 
inportant role in accounting for the variance in UTIL. 

conclusions 

Addressing each of the objectives in turn, the following conclusions 
were drawn. 

* ADMIN* had no direct effect on administrators' readiness for 
utilization (UTIL). The involvement of clients (aiENT) in the evaluation 
process had the greatest effect on administrators' readiness for utilization 
of the results when administrators were not involved personally. A moderate 
level of client involvement as stakeholders appeared to be optimum. 

* Most of the relationships predicted by the model (Figure 1) were 
determined to be lower than expected. The manipulated variables, in 
particular, had negligible or lew correlations with the dependent variables 
STAKE, TECH, QC«L and OTIL. Possible explanatiais were a) the simulated 
conditions for the experiment were not strong enough for the respondents to 
fully uriderstand the nature of the treatments, b) the simulated conditions 
were too unnatural for the participants to respond as they would have 
responded to a real' evaluation report, c) stakeholder involvement cannot 
be sinulated since process rather than outcome is the key aspect of 
participation or d) the level of stakeholder involvement actually has little 
or no effect on perceptions of the report, perceived adequacy of stakeholder 
involvement or readiness for utilization. 

STAKE, TECH and QOT»L were all substantially related to OTIL, leaving 
open the question of what could have affected the respond ents' perceptions 
of these report variables if ADMIN and aiEMT were not major influences. 
Oie conclusion is that an important antecendent variable (possibly: previous 
experience with the evaluation process?) was missing from the study. 

The relationship between ATT and OTIL was statistically significant 
(moderate) only when adninistrators were not involved in the evaluation. 
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Figure 3a.b: Explanation of the Variance in Readiness for Utilization (UTIL) 
by Selected Variables 

- Refers to the contribution of appropriate manipulated variables for each 
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This leads to the conclusion that administrators must be involved as 
stakeholders if evaluators are concerned with overcoming (possibly negative) 
pre-existing attitudes toward evaluation. 

An unexpected finding was that the expected relationship between E\AL 
and ATT was determined to be negligible. Possible explanations include a) 
evaluation training has no direct relationship with the attitude of the 
trainee, but acts through practical experience with evaluation or b) the 
type of evaluation training provided CES may be geared more to cognitive 
than to affective processes. This U)ads to the conclusion that training 
programs that focus on affective aspects of evaluation need to be offered to 
extension personnel. Attitudes may be positively affected if in-service 
training were designed with the following objectives: to increase desire to 
use evaluation as a program development tool, and to increase understanding 
as to the usefulness of evaluation. 

* A significant portion of the variance in readiness for utilization 
was explained by the variables in the model. Overall, 51.4 percent was 
accounted for by the linear combination of CLIENT, CLIENT squared, ATT, 
STAKE, QUAL and TECH. The relative importance of the contribution of the 
variables, however, was related to the level of administrator involvement as 
stakeholders. An interaction of f^rts was found, since the relative 
importaiice of CLIENT and ATT decreased, while the relative importance of 
STAKE increased as the level of admin increased (Figure 3a, b) . The 
conclusion is that the merits of the report will have more wei^t if 
administrators are included in the evaluation process. If administrators 
are excluded, the involvement of clients can have a small positive effect on 
OTIL, but the contribution of ATT increases greatly, leading to possible 
problems with poor pre-axisting attitudes toward evaluation. 

Recomnendations 

The first set of recommendations were developed for practitioners of 
evaluation ; 

* Both administrators and clients should be included as stakeholders 
in the evaluation process in order to maximize the chance that the 
administrator receiving the report will react to qualities of the report 
itself rather than depending on his/her pre-existing attitude toward 
evaluation. 

* Adninistrators should be involved at the highest level possible 
within their time constraints. As the level of adninistrator involvement 
increases, the influence of pre-existing attitude on readiness for 
utilization of the results decreases. 

* Clients should be involved at a moderate level, especially if 
administrators are not involvjd as stakeholders. Inclusion of clients at 
the highest level can lead to decreased readiness for utilization due to 
increased expectations on the part of administrators. If clients are 
included at the highest level, evaluators must oe willing to take extra 
care in reporting on the role of the stakeholdets in the evaluation p-ocess. 
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Hie second group of recommendations were developed for extension 
educators w ho specialize in evaluation: 



* Evaluation education programs (e.g. in-service training programs, 
workshops) offered to CES personnel should be redesigned to focus more on 
the why to do* as opposed to the ^how to do* evaVjation. Greater emphasis 
on the affective realm of evaluation may lead to improve-? attitudes toward 
evaluation. Attitude was shown in this study to be positively related to 
readiness for utilization of evaluation results. 

* Evaluation education programs offered to CES personnel should 
include a) information regarding the purpose and benefits of meaningful 
involvement of stakeholders and b) information regarding how best to include 
stakeholders in the evaluation process. The complex nature of stakeholder 
approaches requires careful training and practice for expected benefits to 
resuli.. 

The final set of recommendations were developed for researchers in the 
field of evaluation: 

* Although the report variables (STAKE, QUAL, TECH) were shown to 
explain nuch of the variance in readiness for utilization in this study, the 
variances in STAKE, TECH and QUAL were not explained by the manipulated 
variables ADMIN and aiEMT, despite the experimental nature of the study, a 
question that should be addressed by further reseai'ch is: What are other 
possible variable{s) that might contribute to explaining the variance in 
these report variables? 

f^^ho^h 3 respectable portion of the variance in readiness for 
utilization was explained in the study, nearly one third of the variance 
remains unexplained. Future research should attenpt to unveil additional 
variables of interest. One possibility arising from this study is the 
variable previous experience with the evaluation process'. 

* Investigation of the role of the involvement of stakeholders in the 
evaluation process may be too complex to simulate as was attempted in this 
study. Future research in this area should use more natural conditions for 
controlled studies, perhaps b^ involving administrators more actively by 
using role playing as opposed to written treatments. 
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